Prospective validation of a seizure diary forecasting falls short.

OBJECTIVE: Recently, a deep learning artificial intelligence (AI) model forecasted seizure risk using retrospective seizure diaries with higher accuracy than random forecasts. The present study sought to prospectively evaluate the same algorithm. METHODS: We recruited a prospective cohort of 46 people with epilepsy; 25 completed sufficient data entry for analysis (median = 5 months). We used the same AI method as in our prior study. Group-level and individual-level Brier Skill Scores (BSSs) compared random forecasts and simple moving average forecasts to the AI. RESULTS: The AI had an area under the receiver operating characteristic curve of .82. At the group level, the AI outperformed random forecasting (BSS = .53). At the individual level, AI outperformed random in 28% of cases. At the group and individual level, the moving average outperformed the AI. If pre-enrollment (nonverified) diaries (with presumed underreporting) were included, the AI significantly outperformed both comparators. Surveys showed most did not mind poor-quality LOW-RISK or HIGH-RISK forecasts, yet 91% wanted access to these forecasts. SIGNIFICANCE: The previously developed AI forecasting tool did not outperform a very simple moving average forecasting in this prospective cohort, suggesting that the AI model should be replaced.

Large Language Model Performance on Practice Epilepsy Board Examinations.

This diagnostic study examines whether large language models are able to pass practice licensing examinations for epilepsy.

Inductive reasoning with large language models: a simulated randomized controlled trial for epilepsy.

Importance: The analysis of electronic medical records at scale to learn from clinical experience is currently very challenging. The integration of artificial intelligence (AI), specifically foundational large language models (LLMs), into an analysis pipeline may overcome some of the current limitations of modest input sizes, inaccuracies, biases, and incomplete knowledge bases. Objective: To explore the effectiveness of using an LLM for generating realistic clinical data and other LLMs for summarizing and synthesizing information in a model system, simulating a randomized clinical trial (RCT) in epilepsy to demonstrate the potential of inductive reasoning via medical chart review. Design: An LLM-generated simulated RCT based on a RCT for treatment with an antiseizure medication, cenobamate, including a placebo arm and a full-strength drug arm, evaluated by an LLM-based pipeline versus a human reader. Setting: Simulation based on realistic seizure diaries, treatment effects, reported symptoms and clinical notes generated by LLMs with multiple different neurologist writing styles. Participants: Simulated cohort of 240 patients, divided 1:1 into placebo and drug arms. Intervention: Utilization of LLMs for the generation of clinical notes and for the synthesis of data from these notes, aiming to evaluate the efficacy and safety of cenobamate in seizure control either with a human evaluator or AI-pipeline. Measures: The AI and human analysis focused on identifying the number of seizures, symptom reports, and treatment efficacy, with statistical analysis comparing the 50%-responder rate and median percentage change between the placebo and drug arms, as well as side effect rates in each arm. Results: AI closely mirrored human analysis, demonstrating the drug's efficacy with marginal differences (<3%) in identifying both drug efficacy and reported symptoms. Conclusions and Relevance: This study showcases the potential of LLMs accurately simulate and analyze clinical trials. Significantly, it highlights the ability of LLMs to reconstruct essential trial elements, identify treatment effects, and recognize reported symptoms, within a realistic clinical framework. The findings underscore the relevance of LLMs in future clinical research, offering a scalable, efficient alternative to traditional data mining methods without the need for specialized medical language training.

Minimum clinical utility standards for wearable seizure detectors: A simulation study.

OBJECTIVE: Epilepsy management employs self-reported seizure diaries, despite evidence of seizure underreporting. Wearable and implantable seizure detection devices are now becoming more widely available. There are no clear guidelines about what levels of accuracy are sufficient. This study aimed to simulate clinical use cases and identify the necessary level of accuracy for each. METHODS: Using a realistic seizure simulator (CHOCOLATES), a ground truth was produced, which was then sampled to generate signals from simulated seizure detectors of various capabilities. Five use cases were evaluated: (1) randomized clinical trials (RCTs), (2) medication adjustment in clinic, (3) injury prevention, (4) sudden unexpected death in epilepsy (SUDEP) prevention, and (5) treatment of seizure clusters. We considered sensitivity (0%-100%), false alarm rate (FAR; 0-2/day), and device type (external wearable vs. implant) in each scenario. RESULTS: The RCT case was efficient for a wide range of wearable parameters, though implantable devices were preferred. Lower accuracy wearables resulted in subtle changes in the distribution of patients enrolled in RCTs, and therefore higher sensitivity and lower FAR values were preferred. In the clinic case, a wide range of sensitivity, FAR, and device type yielded similar results. For injury prevention, SUDEP prevention, and seizure cluster treatment, each scenario required high sensitivity and yet was minimally influenced by FAR. SIGNIFICANCE: The choice of use case is paramount in determining acceptable accuracy levels for a wearable seizure detection device. We offer simulation results for determining and verifying utility for specific use case and specific wearable parameters.

Prospective validation of a seizure diary forecasting falls short.

OBJECTIVE: Recently, a deep learning AI model forecasted seizure risk using retrospective seizure diaries with higher accuracy than random forecasts. The present study sought to prospectively evaluate the same algorithm. METHODS: We recruited a prospective cohort of 46 people with epilepsy; 25 completed sufficient data entry for analysis (median 5 months). We used the same AI method as in our prior study. Group-level and individual-level Brier Skill Scores (BSS) compared random forecasts and simple moving average forecasts to the AI. RESULTS: The AI had an AUC of 0.82. At the group level, the AI outperformed random forecasting (BSS=0.53). At the individual level, AI outperformed random in 28% of cases. At the group and individual level, the moving average outperformed the AI. If pre-enrollment (non-verified) diaries (with presumed under-reporting) were included, the AI significantly outperformed both comparators. Surveys showed most did not mind poor quality LOW-RISK or HIGH-RISK forecasts, yet 91% wanted access to these forecasts. SIGNIFICANCE: The previously developed AI forecasting tool did not outperform a very simple moving average forecasting this prospective cohort, suggesting that the AI model should be replaced.

Quantifying and controlling the impact of regression to the mean on randomized controlled trials in epilepsy.

OBJECTIVE: Randomized controlled trials (RCTs) in epilepsy for drug treatments are plagued by high costs. One potential remedy is to reduce placebo response via better control over regression to the mean (RTM). Here, RTM represents an initial observed seizure rate higher than the long-term average, which gradually settles closer to the average, resulting in apparent response to treatment. This study used simulation to clarify the relationship between eligibility criteria and RTM. METHODS: Using a statistically realistic seizure diary simulator, the impact of RTM on placebo response and trial efficacy was explored by varying eligibility criteria for a traditional treatment phase II/III RCT for drug-resistant epilepsy. RESULTS: When the baseline period was included in the eligibility criteria, increasingly larger fractions of RTM were observed (25%-47% vs. 23%-25%). Higher fractions of RTM corresponded with higher expected placebo responses (50% responder rate [RR50]: 2%-9% vs. 0%-8%) and lower statistical efficacy (RR50: 47%-67% vs. 47%-81%). The exclusion of baseline from eligibility criteria was shown to decrease the number of patients needed by roughly 30%. SIGNIFICANCE: The manipulation of eligibility criteria for RCTs has a predictable and important impact on RTM, and therefore on placebo response; the difference between drug and placebo was more easily detected. This in turn impacts trial efficacy and therefore cost. This study found dramatic improvements in efficacy and cost when baseline was not included in eligibility.

Gaps in care following first time seizure in an underserved region: A retrospective analysis.

PURPOSE: This study investigated the characteristics of patients presenting with the first-time seizure (FTS) and whether neurology follow-up occurred in a medically underserved area. METHODS: A retrospective study of adults with a FTS discharged from the Emergency Department (ED) at Loma Linda University between January 1, 2017 and December 31, 2018 was performed. The primary outcome was days from the ED visit to the first neurology visit. Secondary outcomes included repeat ED visits, percentage of patients who had specialty assessment in one year, type of neurologist seen, and percentage lost to follow-up. RESULTS: Of the 1327 patients screened, 753 encounters met criteria for manual review, and after exclusion criteria were applied, 66 unique encounters were eligible. Only 30% of FTS patients followed up with a neurologist. The median duration for neurology follow-up was 92 days (range=5-1180). After initial ED visit, 20% of follow-up patients were diagnosed with epilepsy within 189 days, and 20% of patients re-presented to the ED with recurrent seizures while awaiting their initial neurology appointment. Reasons for lack of follow-up included: referral issues, missed appointments, and shortage of available neurologists. CONCLUSION: This study highlights the significant treatment gap that a first-time seizure clinic (FTSC) could fill in underserved communities. FTSC may reduce the morbidity and mortality associated with untreated recurrent seizures.

Sample Size Analysis for Machine Learning Clinical Validation Studies.

BACKGROUND: Before integrating new machine learning (ML) into clinical practice, algorithms must undergo validation. Validation studies require sample size estimates. Unlike hypothesis testing studies seeking a p-value, the goal of validating predictive models is obtaining estimates of model performance. There is no standard tool for determining sample size estimates for clinical validation studies for machine learning models. METHODS: Our open-source method, Sample Size Analysis for Machine Learning (SSAML) was described and was tested in three previously published models: brain age to predict mortality (Cox Proportional Hazard), COVID hospitalization risk prediction (ordinal regression), and seizure risk forecasting (deep learning). RESULTS: Minimum sample sizes were obtained in each dataset using standardized criteria. DISCUSSION: SSAML provides a formal expectation of precision and accuracy at a desired confidence level. SSAML is open-source and agnostic to data type and ML model. It can be used for clinical validation studies of ML models.

Does deidentification of data from wearable devices give us a false sense of security? A systematic review.

Wearable devices have made it easier to generate and share data collected on individuals. This systematic review seeks to investigate whether deidentifying data from wearable devices is sufficient to protect the privacy of individuals in datasets. We searched Web of Science, IEEE Xplore Digital Library, PubMed, Scopus, and the ACM Digital Library on Dec 6, 2021 (PROSPERO registration number CRD42022312922). We also performed manual searches in journals of interest until April 12, 2022. Although our search strategy had no language restrictions, all retrieved studies were in English. We included studies showing reidentification, identification, or authentication with data from wearable devices. Our search retrieved 17 625 studies, and 72 studies met our inclusion criteria. We designed a custom assessment tool for study quality and risk of bias assessments. 64 studies were classified as high quality and eight as moderate quality, and we did not detect any bias in any of the included studies. Correct identification rates were typically 86-100%, indicating a high risk of reidentification. Additionally, as little as 1-300 s of recording were required to enable reidentification from sensors that are generally not thought to generate identifiable information, such as electrocardiograms. These findings call for concerted efforts to rethink methods for data sharing to promote advances in research innovation while preventing the loss of individual privacy.

Flexible realistic simulation of seizure occurrence recapitulating statistical properties of seizure diaries.

OBJECTIVE: A realistic seizure diary simulator is currently unavailable for many research needs, including clinical trial analysis and evaluation of seizure detection and seizure-forecasting tools. In recent years, important statistical features of seizure diaries have been characterized. These include (1) heterogeneity of individual seizure frequencies, (2) the relation between average seizure rate and standard deviation, (3) multiple risk cycles, (4) seizure clusters, and (5) limitations on inter-seizure intervals. The present study unifies these features into a single model. METHODS: Our approach, Cyclic Heterogeneous Overdispersed Clustered Open-source L-relationship Adjustable Temporally limited E-diary Simulator (CHOCOLATES) is based on a hierarchical model centered on a gamma Poisson generator with several modifiers. This model accounts for the aforementioned statistical properties. The model was validated by simulating 10 000 randomized clinical trials (RCTs) of medication to compare with 23 historical RCTs. Metrics of 50% responder rate (RR50) and median percent change (MPC) were evaluated. We also used CHOCOLATES as input to a seizure-forecasting tool to test the flexibility of the model. We examined the area under the receiver-operating characteristic (ROC) curve (AUC) for test data with and without cycles and clusters. RESULTS: The model recapitulated typical findings in 23 historical RCTs without the necessity of introducing an additional "placebo effect." The model produced the following RR50 values: placebo: 17 ± 4%; drug 38 ± 5%; and the following MPC values: placebo: 13 ± 6%; drug 40 ± 4%. These values are similar to historical data: for RR50: placebo, 21 ± 10%, drug: 43 ± 13%; and for MPC: placebo: 17 ± 10%, drug: 41 ± 11%. The seizure forecasts achieved an AUC of 0.68 with cycles and clusters, whereas without them the AUC was 0.51. SIGNIFICANCE: CHOCOLATES represents the most realistic seizure occurrence simulator to date, based on observations from thousands of patients in different contexts. This tool is open source and flexible, and can be used for many applications, including clinical trial simulation and testing of seizure-forecasting tools.

Exercise, medication adherence, and the menstrual cycle: How much do these change seizure risk?

People with epilepsy can experience tremendous stress from the uncertainty of when a seizure will occur. Three factors deemed important because of their potential influence on seizure risk are exercise, medication adherence, and the menstrual cycle. A narrative review was conducted through PubMed searching for relevant articles on how seizure risk is modified by 1) exercise, 2) medication adherence, and 3) the menstrual cycle. There was no consensus about the impact of exercise on seizure risk. Studies about medication nonadherence suggested an increase in seizure risk, but there was not a sufficient amount of data for a definitive conclusion. Most studies about the menstrual cycle reported an increase in seizures connected to a specific aspect of the menstrual cycle. No definitive studies were available to quantify this impact precisely. All three triggers reviewed had gaps in the research available, making it not yet possible to definitively quantify a relationship to seizure risk. More quantitative prospective studies are needed to ascertain the extent to which these triggers modify seizure risk.

Focal Cooling for Drug-Resistant Epilepsy: A Review.

Importance: Epilepsy affects at least 1.2% of the population, with one-third of cases considered to be drug-resistant epilepsy (DRE). For these cases, focal cooling therapy may be a potential avenue for treatment, offering hope to people with DRE for freedom from seizure. The therapy leverages neuroscience and engineering principles to deliver a reversible treatment unhindered by pharmacology. Observations: Analogous to (but safer than) the use of global cooling in postcardiac arrest and neonatal ischemic injury, extensive research supports the premise that focal cooling as a long-term treatment for epilepsy could be effective. The potential advantages of focal cooling are trifold: stopping epileptiform discharges, seizures, and status epilepticus safely across species (including humans). Conclusions and Relevance: This Review presents the most current evidence supporting focal cooling in epilepsy. Cooling has been demonstrated as a potentially safe and effective treatment modality for DRE, although it is not yet ready for use in humans outside of randomized clinical trials. The Review will also offer a brief overview of the technical challenges related to focal cooling in humans, including the optimal device design and cooling parameters.

Can machine learning improve randomized clinical trial analysis?

PURPOSE: Recently a realistic simulator of patient seizure diaries was developed that can reproduce effects seen in randomized clinical trials (RCTs). RCTs suffer from high costs and statistical inefficiencies. Using realistic simulation and machine learning this study aimed to identify a more statistically efficient outcome metric. METHODS: Five candidate deep learning architectures with 54 permutations of hyperparameters were compared to the traditional standard, median percent change (MPC). Each were also tested for type 1 error. All models had similar outcomes, with appropriate low levels of type 1 error. RESULTS: The simplest model was equivalent to a logistic regression of a histogram of individual percentage changes in seizure rate, requiring 21-22% less patients to discriminate drug from placebo at 90% power. This model was referred to as LPC. CONCLUSION: Future studies to validate LPC may enable faster, cheaper and more efficient clinical trials.

Guidelines for Conducting Ethical Artificial Intelligence Research in Neurology: A Systematic Approach for Clinicians and Researchers.

Preemptive recognition of the ethical implications of study design and algorithm choices in artificial intelligence (AI) research is an important but challenging process. AI applications have begun to transition from a promising future to clinical reality in neurology. As the clinical management of neurology is often concerned with discrete, often unpredictable, and highly consequential events linked to multimodal data streams over long timescales, forthcoming advances in AI have great potential to transform care for patients. However, critical ethical questions have been raised with implementation of the first AI applications in clinical practice. Clearly, AI will have far-reaching potential to promote, but also to endanger, ethical clinical practice. This article employs an anticipatory ethics approach to scrutinize how researchers in neurology can methodically identify ethical ramifications of design choices early in the research and development process, with a goal of preempting unintended consequences that may violate principles of ethical clinical care. First, we discuss the use of a systematic framework for researchers to identify ethical ramifications of various study design and algorithm choices. Second, using epilepsy as a paradigmatic example, anticipatory clinical scenarios that illustrate unintended ethical consequences are discussed, and failure points in each scenario evaluated. Third, we provide practical recommendations for understanding and addressing ethical ramifications early in methods development stages. Awareness of the ethical implications of study design and algorithm choices that may unintentionally enter AI is crucial to ensuring that incorporation of AI into neurology care leads to patient benefit rather than harm.

Patterns of Recording Epileptic Spasms in an Electronic Seizure Diary Compared With Video-EEG and Historical Cohorts.

BACKGROUND: Use of electronic seizure diaries (e-diaries) by caregivers of children with epileptic spasms is not well understood. We describe the demographic and seizure-related information of children with epileptic spasms captured in a widely used e-diary and explore the potential biases in how caregivers report these data. METHODS: We analyzed children with epileptic spasms in an e-diary, Seizure Tracker, from 2007 to 2018. We described variables including sex, time of seizure, percentage of spasms occurring as individual spasms (versus in clusters), cluster duration, and number of spasms per cluster. We compared seizure characteristics in the e-diary cohort with published cohorts to identify biases in caregiver-reported epileptic spasms. We also reviewed seizure patterns in a small cohort of children with epileptic spasms monitored on overnight video-electroencephalography (vEEG). RESULTS: There were 314 children in the e-diary cohort and nine children in the vEEG cohort. The e-diary cohort was more likely than expected to report counts divisible by five. The e-diary cohort had a lower proportion of nighttime spasms than expected based on data from published cohorts. The e-diary cohort had a significantly lower percentage of spasms as individual spasms, a greater number of spasms per cluster, and a greater cluster duration relative to the vEEG cohort. CONCLUSIONS: Caregivers using e-diaries for epileptic spasms may miss individual spams, be more likely to report long clusters, round counts to the nearest five, and underreport nighttime spasms. Clinicians should be aware of these reporting biases when using e-diary data to guide care for children with epileptic spasms.

Electroencephalographic Abnormalities are Common in COVID-19 and are Associated with Outcomes.

OBJECTIVE: The aim was to determine the prevalence and risk factors for electrographic seizures and other electroencephalographic (EEG) patterns in patients with Coronavirus disease 2019 (COVID-19) undergoing clinically indicated continuous electroencephalogram (cEEG) monitoring and to assess whether EEG findings are associated with outcomes. METHODS: We identified 197 patients with COVID-19 referred for cEEG at 9 participating centers. Medical records and EEG reports were reviewed retrospectively to determine the incidence of and clinical risk factors for seizures and other epileptiform patterns. Multivariate Cox proportional hazards analysis assessed the relationship between EEG patterns and clinical outcomes. RESULTS: Electrographic seizures were detected in 19 (9.6%) patients, including nonconvulsive status epilepticus (NCSE) in 11 (5.6%). Epileptiform abnormalities (either ictal or interictal) were present in 96 (48.7%). Preceding clinical seizures during hospitalization were associated with both electrographic seizures (36.4% in those with vs 8.1% in those without prior clinical seizures, odds ratio [OR] 6.51, p = 0.01) and NCSE (27.3% vs 4.3%, OR 8.34, p = 0.01). A pre-existing intracranial lesion on neuroimaging was associated with NCSE (14.3% vs 3.7%; OR 4.33, p = 0.02). In multivariate analysis of outcomes, electrographic seizures were an independent predictor of in-hospital mortality (hazard ratio [HR] 4.07 [1.44-11.51], p < 0.01). In competing risks analysis, hospital length of stay increased in the presence of NCSE (30 day proportion discharged with vs without NCSE: HR 0.21 [0.03-0.33] vs 0.43 [0.36-0.49]). INTERPRETATION: This multicenter retrospective cohort study demonstrates that seizures and other epileptiform abnormalities are common in patients with COVID-19 undergoing clinically indicated cEEG and are associated with adverse clinical outcomes. ANN NEUROL 2021;89:872-883.

Epilepsy during the COVID-19 pandemic lockdown: a US population survey.

OBJECTIVE: This study sought to understand issues facing people with epilepsy (PWE) during the lockdown period of the COVID-19 pandemic in the United States. METHODS: We conducted a cross-sectional study using a 20-question survey that used SeziureTracker.com, sent to eligible PWE and their caregivers on May 6th, 2020. Questions about demographics and medical history were used to calculate COVID mortality risk odds ratios (OR) compared to a low baseline risk group. RESULTS: In total, 505 responses were collected. Of these, 71% reported no change in seizure rates and 25% reported an increase in seizures, which they attributed primarily to disrupted sleep (63%) and decreased exercise (42%). Mortality risks from COVID-19 had median OR of 1.67, ranging 1.00-906.98. Fear about hospitalization (53%) and concern for loved ones (52%) were prominent concerns. Of the respondents, 5% reported stopping or reducing anti-seizure medications due to problems communicating with doctors, access or cost. Lower-risk COVID patients reported more fear of hospitalization (55% versus 38%, p<0.001) and anxiety about medication access (43% versus 28%, p=0.03) compared with higher-risk COVID patients. Increased anxiety was reported in 47%, and increased depression in 28%. Ten percent without generalized convulsions and 8% with did not know anything about epilepsy devices (VNS, RNS, DBS). SIGNIFICANCE: The COVID-19 pandemic presents unique challenges to PWE, including increased seizure rates, problems with access and cost of life-saving medications. Those with lower COVID-19 risk may have been marginalized more than those with higher risk. Efforts to protect PWE during major public health emergencies should take these findings into account.

Recognizing and refuting the myth of tongue swallowing during a seizure.

OBJECTIVE: There is a harmful myth that persists in modern culture that one should place objects into a seizing person's mouth to prevent "swallowing the tongue." Despite expert guidelines against this, the idea remains alive in popular media and public belief. We aimed to investigate the myth's origins and discredit it. METHODS: A medical and popular literature review was conducted for the allusions to "swallowing one's tongue" and practice recommendations for and against placing objects into a seizing person's mouth. Current prevalence of these beliefs and relevant anatomy and physiology were summarised. RESULTS: The first English language allusions to placing objects in a patient's mouth occurred in the mid-19th century, and the first allusions to swallowing one's tongue during a seizure occurred in the late 19th century. By the mid-20th century, it was clear that some were recommending against the practice of placing objects in a patient's mouth to prevent harm. Relatively recent popular literature and film continue to portray incorrect seizure first aid through at least 2013. There is ample modern literature confirming the anatomical impossibility of swallowing one's tongue and confirming the potential harm of putting objects in a patient's mouth. CONCLUSION: One cannot swallow their tongue during a seizure. Foreign objects should not be placed into a seizing person's mouth. We must continue to disseminate these ideas to our patients and colleagues. As neurologists, we have an obligation to champion safe practices for our patients, especially when popular media and culture continue to propagate dangerous ones.